Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: improve state serialization #6563

Merged
merged 8 commits into from
Mar 28, 2024
Merged

fix: improve state serialization #6563

merged 8 commits into from
Mar 28, 2024

Conversation

twoeths
Copy link
Contributor

@twoeths twoeths commented Mar 19, 2024

Motivation

State serialization happens once per epoch in n-historical state so its performance is important

Description

  • New validator type to optimize its value_serializeToBytes performance, this helps make state.validators.serialize() 3.4x faster

before

    ✔ serialize 20000 validators manually                                 773.4012 ops/s    1.292990 ms/op        -      45968 runs   60.0 s
    ✔ serialize 20000 validators from state                               206.3978 ops/s    4.845013 ms/op        -      12271 runs   60.0 s

after

    ✔ serialize 20000 validators manually                                 768.2498 ops/s    1.301660 ms/op        -      45667 runs   60.0 s
    ✔ serialize 20000 validators from state                               694.0964 ops/s    1.440722 ms/op        -      41260 runs   60.0 s
  • use the new serializeToBytes() api of ssz v0.15.1

part of #5968

Copy link

codecov bot commented Mar 19, 2024

Codecov Report

Merging #6563 (121edfe) into unstable (2f6e819) will increase coverage by 0.03%.
Report is 2 commits behind head on unstable.
The diff coverage is 96.25%.

❗ Current head 121edfe differs from pull request most recent head f8c669b. Consider uploading reports for the commit f8c669b to get more accurate results

Additional details and impacted files
@@             Coverage Diff              @@
##           unstable    #6563      +/-   ##
============================================
+ Coverage     61.49%   61.53%   +0.03%     
============================================
  Files           556      557       +1     
  Lines         58895    58954      +59     
  Branches       1856     1859       +3     
============================================
+ Hits          36216    36275      +59     
  Misses        22638    22638              
  Partials         41       41              

validator: ValueOfFields<typeof ValidatorType>
): number {
output.set(validator.pubkey, offset);
offset += 48;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a way we could document those constants? Or maybe assign them to variable so that they are more self-explanatory?

Copy link
Contributor

github-actions bot commented Mar 19, 2024

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: 55d0587 Previous: 59a49ff Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 779.55 us/op 702.22 us/op 1.11
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 153.68 us/op 52.244 us/op 2.94
BLS verify - blst-native 1.5229 ms/op 1.0730 ms/op 1.42
BLS verifyMultipleSignatures 3 - blst-native 3.1600 ms/op 2.2932 ms/op 1.38
BLS verifyMultipleSignatures 8 - blst-native 6.9042 ms/op 5.0727 ms/op 1.36
BLS verifyMultipleSignatures 32 - blst-native 25.832 ms/op 18.630 ms/op 1.39
BLS verifyMultipleSignatures 64 - blst-native 51.684 ms/op 37.374 ms/op 1.38
BLS verifyMultipleSignatures 128 - blst-native 139.38 ms/op 75.284 ms/op 1.85
BLS deserializing 10000 signatures 1.0600 s/op 790.72 ms/op 1.34
BLS deserializing 100000 signatures 10.442 s/op 8.3216 s/op 1.25
BLS verifyMultipleSignatures - same message - 3 - blst-native 1.4762 ms/op 1.1389 ms/op 1.30
BLS verifyMultipleSignatures - same message - 8 - blst-native 1.6625 ms/op 1.2908 ms/op 1.29
BLS verifyMultipleSignatures - same message - 32 - blst-native 2.7215 ms/op 2.0310 ms/op 1.34
BLS verifyMultipleSignatures - same message - 64 - blst-native 3.9172 ms/op 3.8367 ms/op 1.02
BLS verifyMultipleSignatures - same message - 128 - blst-native 7.3616 ms/op 5.0126 ms/op 1.47
BLS aggregatePubkeys 32 - blst-native 29.778 us/op 22.717 us/op 1.31
BLS aggregatePubkeys 128 - blst-native 111.28 us/op 88.836 us/op 1.25
notSeenSlots=1 numMissedVotes=1 numBadVotes=10 75.405 ms/op 49.227 ms/op 1.53
notSeenSlots=1 numMissedVotes=0 numBadVotes=4 69.630 ms/op 43.485 ms/op 1.60
notSeenSlots=2 numMissedVotes=1 numBadVotes=10 50.262 ms/op 26.989 ms/op 1.86
getSlashingsAndExits - default max 197.74 us/op 132.15 us/op 1.50
getSlashingsAndExits - 2k 457.79 us/op 589.16 us/op 0.78
proposeBlockBody type=full, size=empty 6.6529 ms/op 4.5829 ms/op 1.45
isKnown best case - 1 super set check 326.00 ns/op 625.00 ns/op 0.52
isKnown normal case - 2 super set checks 387.00 ns/op 656.00 ns/op 0.59
isKnown worse case - 16 super set checks 335.00 ns/op 645.00 ns/op 0.52
CheckpointStateCache - add get delete 6.0410 us/op 5.8740 us/op 1.03
validate api signedAggregateAndProof - struct 2.8049 ms/op 2.4744 ms/op 1.13
validate gossip signedAggregateAndProof - struct 2.7885 ms/op 2.4112 ms/op 1.16
validate gossip attestation - vc 640000 1.3637 ms/op 1.1962 ms/op 1.14
batch validate gossip attestation - vc 640000 - chunk 32 160.85 us/op 151.33 us/op 1.06
batch validate gossip attestation - vc 640000 - chunk 64 141.28 us/op 136.09 us/op 1.04
batch validate gossip attestation - vc 640000 - chunk 128 133.18 us/op 132.04 us/op 1.01
batch validate gossip attestation - vc 640000 - chunk 256 125.99 us/op 125.97 us/op 1.00
pickEth1Vote - no votes 1.2206 ms/op 888.51 us/op 1.37
pickEth1Vote - max votes 8.3439 ms/op 7.6490 ms/op 1.09
pickEth1Vote - Eth1Data hashTreeRoot value x2048 17.549 ms/op 12.310 ms/op 1.43
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 20.501 ms/op 24.006 ms/op 0.85
pickEth1Vote - Eth1Data fastSerialize value x2048 597.38 us/op 433.08 us/op 1.38
pickEth1Vote - Eth1Data fastSerialize tree x2048 4.3149 ms/op 5.8644 ms/op 0.74
bytes32 toHexString 493.00 ns/op 472.00 ns/op 1.04
bytes32 Buffer.toString(hex) 281.00 ns/op 325.00 ns/op 0.86
bytes32 Buffer.toString(hex) from Uint8Array 420.00 ns/op 437.00 ns/op 0.96
bytes32 Buffer.toString(hex) + 0x 281.00 ns/op 331.00 ns/op 0.85
Object access 1 prop 0.17400 ns/op 0.21000 ns/op 0.83
Map access 1 prop 0.14700 ns/op 0.19300 ns/op 0.76
Object get x1000 7.1380 ns/op 5.3700 ns/op 1.33
Map get x1000 0.73800 ns/op 0.79600 ns/op 0.93
Object set x1000 51.556 ns/op 25.441 ns/op 2.03
Map set x1000 39.467 ns/op 17.722 ns/op 2.23
Return object 10000 times 0.23480 ns/op 0.23720 ns/op 0.99
Throw Error 10000 times 3.8107 us/op 2.7777 us/op 1.37
fastMsgIdFn sha256 / 200 bytes 3.2360 us/op 2.0690 us/op 1.56
fastMsgIdFn h32 xxhash / 200 bytes 278.00 ns/op 328.00 ns/op 0.85
fastMsgIdFn h64 xxhash / 200 bytes 353.00 ns/op 367.00 ns/op 0.96
fastMsgIdFn sha256 / 1000 bytes 11.271 us/op 6.3540 us/op 1.77
fastMsgIdFn h32 xxhash / 1000 bytes 407.00 ns/op 438.00 ns/op 0.93
fastMsgIdFn h64 xxhash / 1000 bytes 405.00 ns/op 417.00 ns/op 0.97
fastMsgIdFn sha256 / 10000 bytes 101.71 us/op 53.976 us/op 1.88
fastMsgIdFn h32 xxhash / 10000 bytes 1.9020 us/op 1.8990 us/op 1.00
fastMsgIdFn h64 xxhash / 10000 bytes 1.2930 us/op 1.2740 us/op 1.01
send data - 1000 256B messages 16.808 ms/op 12.425 ms/op 1.35
send data - 1000 512B messages 27.393 ms/op 14.897 ms/op 1.84
send data - 1000 1024B messages 38.060 ms/op 27.121 ms/op 1.40
send data - 1000 1200B messages 37.500 ms/op 31.509 ms/op 1.19
send data - 1000 2048B messages 46.880 ms/op 38.901 ms/op 1.21
send data - 1000 4096B messages 43.194 ms/op 33.472 ms/op 1.29
send data - 1000 16384B messages 122.73 ms/op 100.11 ms/op 1.23
send data - 1000 65536B messages 483.70 ms/op 381.11 ms/op 1.27
enrSubnets - fastDeserialize 64 bits 1.2840 us/op 1.0150 us/op 1.27
enrSubnets - ssz BitVector 64 bits 406.00 ns/op 456.00 ns/op 0.89
enrSubnets - fastDeserialize 4 bits 163.00 ns/op 204.00 ns/op 0.80
enrSubnets - ssz BitVector 4 bits 413.00 ns/op 446.00 ns/op 0.93
prioritizePeers score -10:0 att 32-0.1 sync 2-0 101.11 us/op 69.105 us/op 1.46
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 125.79 us/op 85.405 us/op 1.47
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 168.58 us/op 120.91 us/op 1.39
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 295.63 us/op 226.64 us/op 1.30
prioritizePeers score 0:0 att 64-1 sync 4-1 327.44 us/op 205.02 us/op 1.60
array of 16000 items push then shift 1.5897 us/op 1.2957 us/op 1.23
LinkedList of 16000 items push then shift 8.8270 ns/op 5.8340 ns/op 1.51
array of 16000 items push then pop 82.258 ns/op 51.289 ns/op 1.60
LinkedList of 16000 items push then pop 8.6260 ns/op 5.6360 ns/op 1.53
array of 24000 items push then shift 2.3627 us/op 1.8761 us/op 1.26
LinkedList of 24000 items push then shift 8.8630 ns/op 5.8270 ns/op 1.52
array of 24000 items push then pop 121.90 ns/op 80.226 ns/op 1.52
LinkedList of 24000 items push then pop 8.5930 ns/op 5.6510 ns/op 1.52
intersect bitArray bitLen 8 5.7550 ns/op 4.5510 ns/op 1.26
intersect array and set length 8 62.242 ns/op 47.854 ns/op 1.30
intersect bitArray bitLen 128 35.156 ns/op 28.601 ns/op 1.23
intersect array and set length 128 850.27 ns/op 676.70 ns/op 1.26
bitArray.getTrueBitIndexes() bitLen 128 1.4750 us/op 1.2680 us/op 1.16
bitArray.getTrueBitIndexes() bitLen 248 2.4250 us/op 2.0620 us/op 1.18
bitArray.getTrueBitIndexes() bitLen 512 4.8170 us/op 3.9410 us/op 1.22
Buffer.concat 32 items 968.00 ns/op 892.00 ns/op 1.09
Uint8Array.set 32 items 2.3250 us/op 1.8630 us/op 1.25
Set add up to 64 items then delete first 4.2187 us/op 1.7379 us/op 2.43
OrderedSet add up to 64 items then delete first 5.3431 us/op 2.6519 us/op 2.01
Set add up to 64 items then delete last 4.5269 us/op 1.9814 us/op 2.28
OrderedSet add up to 64 items then delete last 6.1697 us/op 2.9397 us/op 2.10
Set add up to 64 items then delete middle 4.5825 us/op 1.9803 us/op 2.31
OrderedSet add up to 64 items then delete middle 7.2202 us/op 4.1372 us/op 1.75
Set add up to 128 items then delete first 9.1432 us/op 3.9253 us/op 2.33
OrderedSet add up to 128 items then delete first 11.572 us/op 6.1099 us/op 1.89
Set add up to 128 items then delete last 9.1220 us/op 3.7622 us/op 2.42
OrderedSet add up to 128 items then delete last 12.003 us/op 5.6536 us/op 2.12
Set add up to 128 items then delete middle 9.0644 us/op 3.7523 us/op 2.42
OrderedSet add up to 128 items then delete middle 17.130 us/op 10.505 us/op 1.63
Set add up to 256 items then delete first 18.722 us/op 7.6853 us/op 2.44
OrderedSet add up to 256 items then delete first 23.839 us/op 12.078 us/op 1.97
Set add up to 256 items then delete last 18.241 us/op 7.3913 us/op 2.47
OrderedSet add up to 256 items then delete last 24.417 us/op 11.291 us/op 2.16
Set add up to 256 items then delete middle 17.940 us/op 7.3991 us/op 2.42
OrderedSet add up to 256 items then delete middle 44.710 us/op 30.189 us/op 1.48
transfer serialized Status (84 B) 1.6800 us/op 1.3200 us/op 1.27
copy serialized Status (84 B) 1.2470 us/op 1.0740 us/op 1.16
transfer serialized SignedVoluntaryExit (112 B) 1.8480 us/op 1.4010 us/op 1.32
copy serialized SignedVoluntaryExit (112 B) 1.2950 us/op 1.1060 us/op 1.17
transfer serialized ProposerSlashing (416 B) 2.1100 us/op 2.0570 us/op 1.03
copy serialized ProposerSlashing (416 B) 1.7310 us/op 1.9890 us/op 0.87
transfer serialized Attestation (485 B) 2.0710 us/op 1.9540 us/op 1.06
copy serialized Attestation (485 B) 1.6130 us/op 1.8380 us/op 0.88
transfer serialized AttesterSlashing (33232 B) 2.0860 us/op 2.1090 us/op 0.99
copy serialized AttesterSlashing (33232 B) 4.9790 us/op 4.0480 us/op 1.23
transfer serialized Small SignedBeaconBlock (128000 B) 2.2790 us/op 1.7850 us/op 1.28
copy serialized Small SignedBeaconBlock (128000 B) 13.732 us/op 8.3590 us/op 1.64
transfer serialized Avg SignedBeaconBlock (200000 B) 2.4700 us/op 1.9330 us/op 1.28
copy serialized Avg SignedBeaconBlock (200000 B) 21.924 us/op 12.225 us/op 1.79
transfer serialized BlobsSidecar (524380 B) 2.3200 us/op 2.0250 us/op 1.15
copy serialized BlobsSidecar (524380 B) 84.009 us/op 171.27 us/op 0.49
transfer serialized Big SignedBeaconBlock (1000000 B) 2.6820 us/op 2.5140 us/op 1.07
copy serialized Big SignedBeaconBlock (1000000 B) 165.84 us/op 144.72 us/op 1.15
pass gossip attestations to forkchoice per slot 3.8240 ms/op 2.7276 ms/op 1.40
forkChoice updateHead vc 100000 bc 64 eq 0 697.62 us/op 489.19 us/op 1.43
forkChoice updateHead vc 600000 bc 64 eq 0 5.0245 ms/op 2.7366 ms/op 1.84
forkChoice updateHead vc 1000000 bc 64 eq 0 6.9262 ms/op 4.5229 ms/op 1.53
forkChoice updateHead vc 600000 bc 320 eq 0 4.1240 ms/op 2.5552 ms/op 1.61
forkChoice updateHead vc 600000 bc 1200 eq 0 4.2928 ms/op 2.6651 ms/op 1.61
forkChoice updateHead vc 600000 bc 7200 eq 0 5.3550 ms/op 3.3438 ms/op 1.60
forkChoice updateHead vc 600000 bc 64 eq 1000 11.410 ms/op 9.6986 ms/op 1.18
forkChoice updateHead vc 600000 bc 64 eq 10000 12.536 ms/op 9.7625 ms/op 1.28
forkChoice updateHead vc 600000 bc 64 eq 300000 17.564 ms/op 12.114 ms/op 1.45
computeDeltas 500000 validators 300 proto nodes 6.6123 ms/op 3.1287 ms/op 2.11
computeDeltas 500000 validators 1200 proto nodes 6.7426 ms/op 2.9631 ms/op 2.28
computeDeltas 500000 validators 7200 proto nodes 6.5904 ms/op 3.2632 ms/op 2.02
computeDeltas 750000 validators 300 proto nodes 9.6185 ms/op 4.8526 ms/op 1.98
computeDeltas 750000 validators 1200 proto nodes 9.5816 ms/op 4.7782 ms/op 2.01
computeDeltas 750000 validators 7200 proto nodes 9.9011 ms/op 4.8722 ms/op 2.03
computeDeltas 1400000 validators 300 proto nodes 18.713 ms/op 9.0137 ms/op 2.08
computeDeltas 1400000 validators 1200 proto nodes 18.752 ms/op 9.1248 ms/op 2.06
computeDeltas 1400000 validators 7200 proto nodes 19.193 ms/op 9.0139 ms/op 2.13
computeDeltas 2100000 validators 300 proto nodes 29.195 ms/op 12.666 ms/op 2.31
computeDeltas 2100000 validators 1200 proto nodes 30.616 ms/op 12.957 ms/op 2.36
computeDeltas 2100000 validators 7200 proto nodes 30.940 ms/op 13.313 ms/op 2.32
altair processAttestation - 250000 vs - 7PWei normalcase 3.3984 ms/op 1.6027 ms/op 2.12
altair processAttestation - 250000 vs - 7PWei worstcase 5.0015 ms/op 2.3175 ms/op 2.16
altair processAttestation - setStatus - 1/6 committees join 208.65 us/op 98.768 us/op 2.11
altair processAttestation - setStatus - 1/3 committees join 383.94 us/op 200.85 us/op 1.91
altair processAttestation - setStatus - 1/2 committees join 536.80 us/op 281.44 us/op 1.91
altair processAttestation - setStatus - 2/3 committees join 702.66 us/op 375.87 us/op 1.87
altair processAttestation - setStatus - 4/5 committees join 1.0133 ms/op 491.76 us/op 2.06
altair processAttestation - setStatus - 100% committees join 1.4038 ms/op 580.77 us/op 2.42
altair processBlock - 250000 vs - 7PWei normalcase 10.122 ms/op 7.2186 ms/op 1.40
altair processBlock - 250000 vs - 7PWei normalcase hashState 38.453 ms/op 30.114 ms/op 1.28
altair processBlock - 250000 vs - 7PWei worstcase 57.227 ms/op 30.940 ms/op 1.85
altair processBlock - 250000 vs - 7PWei worstcase hashState 129.46 ms/op 90.184 ms/op 1.44
phase0 processBlock - 250000 vs - 7PWei normalcase 3.6605 ms/op 1.8863 ms/op 1.94
phase0 processBlock - 250000 vs - 7PWei worstcase 37.872 ms/op 23.704 ms/op 1.60
altair processEth1Data - 250000 vs - 7PWei normalcase 827.17 us/op 368.48 us/op 2.24
getExpectedWithdrawals 250000 eb:1,eth1:1,we:0,wn:0,smpl:15 21.669 us/op 9.0520 us/op 2.39
getExpectedWithdrawals 250000 eb:0.95,eth1:0.1,we:0.05,wn:0,smpl:219 71.766 us/op 53.013 us/op 1.35
getExpectedWithdrawals 250000 eb:0.95,eth1:0.3,we:0.05,wn:0,smpl:42 43.476 us/op 17.918 us/op 2.43
getExpectedWithdrawals 250000 eb:0.95,eth1:0.7,we:0.05,wn:0,smpl:18 22.560 us/op 8.0010 us/op 2.82
getExpectedWithdrawals 250000 eb:0.1,eth1:0.1,we:0,wn:0,smpl:1020 279.74 us/op 101.56 us/op 2.75
getExpectedWithdrawals 250000 eb:0.03,eth1:0.03,we:0,wn:0,smpl:11777 1.9409 ms/op 1.0578 ms/op 1.83
getExpectedWithdrawals 250000 eb:0.01,eth1:0.01,we:0,wn:0,smpl:16384 2.5345 ms/op 1.0323 ms/op 2.46
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,smpl:16384 2.1783 ms/op 1.4796 ms/op 1.47
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,nocache,smpl:16384 4.5522 ms/op 2.3059 ms/op 1.97
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,smpl:16384 3.0594 ms/op 1.5118 ms/op 2.02
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,nocache,smpl:16384 7.1334 ms/op 3.4643 ms/op 2.06
Tree 40 250000 create 942.20 ms/op 249.83 ms/op 3.77
Tree 40 250000 get(125000) 231.03 ns/op 111.66 ns/op 2.07
Tree 40 250000 set(125000) 3.0615 us/op 748.71 ns/op 4.09
Tree 40 250000 toArray() 29.269 ms/op 9.1599 ms/op 3.20
Tree 40 250000 iterate all - toArray() + loop 30.455 ms/op 9.3845 ms/op 3.25
Tree 40 250000 iterate all - get(i) 87.924 ms/op 37.441 ms/op 2.35
MutableVector 250000 create 20.819 ms/op 11.616 ms/op 1.79
MutableVector 250000 get(125000) 7.3740 ns/op 5.6400 ns/op 1.31
MutableVector 250000 set(125000) 765.68 ns/op 204.44 ns/op 3.75
MutableVector 250000 toArray() 5.1657 ms/op 2.3744 ms/op 2.18
MutableVector 250000 iterate all - toArray() + loop 5.5083 ms/op 2.5834 ms/op 2.13
MutableVector 250000 iterate all - get(i) 1.7450 ms/op 1.2848 ms/op 1.36
Array 250000 create 5.7332 ms/op 2.3550 ms/op 2.43
Array 250000 clone - spread 7.0975 ms/op 1.1757 ms/op 6.04
Array 250000 get(125000) 3.6460 ns/op 1.0650 ns/op 3.42
Array 250000 set(125000) 7.0580 ns/op 1.2760 ns/op 5.53
Array 250000 iterate all - loop 214.13 us/op 152.42 us/op 1.40
effectiveBalanceIncrements clone Uint8Array 300000 107.42 us/op 14.953 us/op 7.18
effectiveBalanceIncrements clone MutableVector 300000 2.6860 us/op 447.00 ns/op 6.01
effectiveBalanceIncrements rw all Uint8Array 300000 233.32 us/op 182.38 us/op 1.28
effectiveBalanceIncrements rw all MutableVector 300000 282.77 ms/op 66.367 ms/op 4.26
phase0 afterProcessEpoch - 250000 vs - 7PWei 132.13 ms/op 76.436 ms/op 1.73
phase0 beforeProcessEpoch - 250000 vs - 7PWei 72.566 ms/op 42.630 ms/op 1.70
altair processEpoch - mainnet_e81889 876.96 ms/op 404.27 ms/op 2.17
mainnet_e81889 - altair beforeProcessEpoch 170.99 ms/op 67.733 ms/op 2.52
mainnet_e81889 - altair processJustificationAndFinalization 39.619 us/op 10.550 us/op 3.76
mainnet_e81889 - altair processInactivityUpdates 14.123 ms/op 4.0237 ms/op 3.51
mainnet_e81889 - altair processRewardsAndPenalties 144.08 ms/op 63.182 ms/op 2.28
mainnet_e81889 - altair processRegistryUpdates 7.8860 us/op 1.5430 us/op 5.11
mainnet_e81889 - altair processSlashings 1.9690 us/op 546.00 ns/op 3.61
mainnet_e81889 - altair processEth1DataReset 2.4150 us/op 561.00 ns/op 4.30
mainnet_e81889 - altair processEffectiveBalanceUpdates 9.8901 ms/op 1.7538 ms/op 5.64
mainnet_e81889 - altair processSlashingsReset 13.292 us/op 2.2460 us/op 5.92
mainnet_e81889 - altair processRandaoMixesReset 14.346 us/op 2.5420 us/op 5.64
mainnet_e81889 - altair processHistoricalRootsUpdate 1.7210 us/op 683.00 ns/op 2.52
mainnet_e81889 - altair processParticipationFlagUpdates 6.9590 us/op 1.6330 us/op 4.26
mainnet_e81889 - altair processSyncCommitteeUpdates 2.1700 us/op 524.00 ns/op 4.14
mainnet_e81889 - altair afterProcessEpoch 157.75 ms/op 81.552 ms/op 1.93
capella processEpoch - mainnet_e217614 2.4333 s/op 1.7112 s/op 1.42
mainnet_e217614 - capella beforeProcessEpoch 534.15 ms/op 387.16 ms/op 1.38
mainnet_e217614 - capella processJustificationAndFinalization 19.152 us/op 7.4610 us/op 2.57
mainnet_e217614 - capella processInactivityUpdates 22.777 ms/op 16.184 ms/op 1.41
mainnet_e217614 - capella processRewardsAndPenalties 775.97 ms/op 443.90 ms/op 1.75
mainnet_e217614 - capella processRegistryUpdates 36.956 us/op 24.923 us/op 1.48
mainnet_e217614 - capella processSlashings 1.2580 us/op 622.00 ns/op 2.02
mainnet_e217614 - capella processEth1DataReset 1.1330 us/op 666.00 ns/op 1.70
mainnet_e217614 - capella processEffectiveBalanceUpdates 5.3067 ms/op 3.5063 ms/op 1.51
mainnet_e217614 - capella processSlashingsReset 8.6520 us/op 2.6990 us/op 3.21
mainnet_e217614 - capella processRandaoMixesReset 8.3120 us/op 5.3110 us/op 1.57
mainnet_e217614 - capella processHistoricalRootsUpdate 1.1550 us/op 474.00 ns/op 2.44
mainnet_e217614 - capella processParticipationFlagUpdates 4.2170 us/op 1.2230 us/op 3.45
mainnet_e217614 - capella afterProcessEpoch 349.12 ms/op 208.28 ms/op 1.68
phase0 processEpoch - mainnet_e58758 548.11 ms/op 367.95 ms/op 1.49
mainnet_e58758 - phase0 beforeProcessEpoch 138.18 ms/op 104.41 ms/op 1.32
mainnet_e58758 - phase0 processJustificationAndFinalization 24.897 us/op 14.108 us/op 1.76
mainnet_e58758 - phase0 processRewardsAndPenalties 78.629 ms/op 58.003 ms/op 1.36
mainnet_e58758 - phase0 processRegistryUpdates 23.227 us/op 8.5600 us/op 2.71
mainnet_e58758 - phase0 processSlashings 2.0750 us/op 565.00 ns/op 3.67
mainnet_e58758 - phase0 processEth1DataReset 1.3920 us/op 568.00 ns/op 2.45
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 1.9286 ms/op 806.72 us/op 2.39
mainnet_e58758 - phase0 processSlashingsReset 8.6690 us/op 2.3460 us/op 3.70
mainnet_e58758 - phase0 processRandaoMixesReset 12.066 us/op 3.8630 us/op 3.12
mainnet_e58758 - phase0 processHistoricalRootsUpdate 1.4200 us/op 588.00 ns/op 2.41
mainnet_e58758 - phase0 processParticipationRecordUpdates 9.8620 us/op 3.4800 us/op 2.83
mainnet_e58758 - phase0 afterProcessEpoch 121.52 ms/op 68.290 ms/op 1.78
phase0 processEffectiveBalanceUpdates - 250000 normalcase 2.1055 ms/op 1.0074 ms/op 2.09
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 2.1391 ms/op 1.1336 ms/op 1.89
altair processInactivityUpdates - 250000 normalcase 36.148 ms/op 21.702 ms/op 1.67
altair processInactivityUpdates - 250000 worstcase 38.740 ms/op 25.116 ms/op 1.54
phase0 processRegistryUpdates - 250000 normalcase 24.842 us/op 10.234 us/op 2.43
phase0 processRegistryUpdates - 250000 badcase_full_deposits 666.07 us/op 407.61 us/op 1.63
phase0 processRegistryUpdates - 250000 worstcase 0.5 199.44 ms/op 108.79 ms/op 1.83
altair processRewardsAndPenalties - 250000 normalcase 97.541 ms/op 71.084 ms/op 1.37
altair processRewardsAndPenalties - 250000 worstcase 85.995 ms/op 58.030 ms/op 1.48
phase0 getAttestationDeltas - 250000 normalcase 15.970 ms/op 5.6628 ms/op 2.82
phase0 getAttestationDeltas - 250000 worstcase 16.456 ms/op 5.8396 ms/op 2.82
phase0 processSlashings - 250000 worstcase 144.19 us/op 89.160 us/op 1.62
altair processSyncCommitteeUpdates - 250000 198.59 ms/op 110.90 ms/op 1.79
BeaconState.hashTreeRoot - No change 843.00 ns/op 401.00 ns/op 2.10
BeaconState.hashTreeRoot - 1 full validator 191.53 us/op 95.098 us/op 2.01
BeaconState.hashTreeRoot - 32 full validator 1.9516 ms/op 1.1800 ms/op 1.65
BeaconState.hashTreeRoot - 512 full validator 27.072 ms/op 13.236 ms/op 2.05
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 232.73 us/op 202.53 us/op 1.15
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 3.7721 ms/op 2.5664 ms/op 1.47
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 44.911 ms/op 32.653 ms/op 1.38
BeaconState.hashTreeRoot - 1 balances 206.19 us/op 132.72 us/op 1.55
BeaconState.hashTreeRoot - 32 balances 1.7650 ms/op 1.2885 ms/op 1.37
BeaconState.hashTreeRoot - 512 balances 16.902 ms/op 13.275 ms/op 1.27
BeaconState.hashTreeRoot - 250000 balances 284.82 ms/op 220.65 ms/op 1.29
aggregationBits - 2048 els - zipIndexesInBitList 50.661 us/op 28.187 us/op 1.80
byteArrayEquals 32 90.661 ns/op 63.299 ns/op 1.43
Buffer.compare 32 65.396 ns/op 35.853 ns/op 1.82
byteArrayEquals 1024 2.3756 us/op 1.7071 us/op 1.39
Buffer.compare 1024 94.280 ns/op 43.227 ns/op 2.18
byteArrayEquals 16384 37.507 us/op 26.886 us/op 1.40
Buffer.compare 16384 274.05 ns/op 223.80 ns/op 1.22
byteArrayEquals 123687377 316.89 ms/op 207.56 ms/op 1.53
Buffer.compare 123687377 10.584 ms/op 3.9913 ms/op 2.65
byteArrayEquals 32 - diff last byte 88.527 ns/op 61.177 ns/op 1.45
Buffer.compare 32 - diff last byte 68.759 ns/op 37.759 ns/op 1.82
byteArrayEquals 1024 - diff last byte 2.4570 us/op 1.7043 us/op 1.44
Buffer.compare 1024 - diff last byte 89.387 ns/op 46.105 ns/op 1.94
byteArrayEquals 16384 - diff last byte 38.458 us/op 27.228 us/op 1.41
Buffer.compare 16384 - diff last byte 314.22 ns/op 217.83 ns/op 1.44
byteArrayEquals 123687377 - diff last byte 302.94 ms/op 198.89 ms/op 1.52
Buffer.compare 123687377 - diff last byte 10.312 ms/op 5.3550 ms/op 1.93
byteArrayEquals 32 - random bytes 7.7370 ns/op 4.5880 ns/op 1.69
Buffer.compare 32 - random bytes 69.597 ns/op 38.711 ns/op 1.80
byteArrayEquals 1024 - random bytes 7.0160 ns/op 4.2930 ns/op 1.63
Buffer.compare 1024 - random bytes 69.500 ns/op 34.313 ns/op 2.03
byteArrayEquals 16384 - random bytes 7.2240 ns/op 4.2980 ns/op 1.68
Buffer.compare 16384 - random bytes 67.212 ns/op 34.966 ns/op 1.92
byteArrayEquals 123687377 - random bytes 13.220 ns/op 7.7700 ns/op 1.70
Buffer.compare 123687377 - random bytes 85.070 ns/op 38.470 ns/op 2.21
regular array get 100000 times 50.053 us/op 39.840 us/op 1.26
wrappedArray get 100000 times 50.284 us/op 39.454 us/op 1.27
arrayWithProxy get 100000 times 17.210 ms/op 9.9750 ms/op 1.73
ssz.Root.equals 61.733 ns/op 53.394 ns/op 1.16
byteArrayEquals 60.988 ns/op 51.928 ns/op 1.17
Buffer.compare 15.698 ns/op 9.1380 ns/op 1.72
shuffle list - 16384 els 9.5197 ms/op 5.7688 ms/op 1.65
shuffle list - 250000 els 142.97 ms/op 81.578 ms/op 1.75
processSlot - 1 slots 21.655 us/op 11.472 us/op 1.89
processSlot - 32 slots 4.3019 ms/op 2.2436 ms/op 1.92
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 65.468 ms/op 52.021 ms/op 1.26
getCommitteeAssignments - req 1 vs - 250000 vc 3.0235 ms/op 2.3972 ms/op 1.26
getCommitteeAssignments - req 100 vs - 250000 vc 4.3197 ms/op 3.5289 ms/op 1.22
getCommitteeAssignments - req 1000 vs - 250000 vc 4.6208 ms/op 3.8139 ms/op 1.21
findModifiedValidators - 10000 modified validators 458.32 ms/op 397.34 ms/op 1.15
findModifiedValidators - 1000 modified validators 308.44 ms/op 325.29 ms/op 0.95
findModifiedValidators - 100 modified validators 291.74 ms/op 316.90 ms/op 0.92
findModifiedValidators - 10 modified validators 267.04 ms/op 320.17 ms/op 0.83
findModifiedValidators - 1 modified validators 270.02 ms/op 290.97 ms/op 0.93
findModifiedValidators - no difference 261.76 ms/op 319.64 ms/op 0.82
compare ViewDUs 5.6516 s/op 3.9119 s/op 1.44
compare each validator Uint8Array 1.8768 s/op 1.5831 s/op 1.19
compare ViewDU to Uint8Array 1.5710 s/op 843.24 ms/op 1.86
migrate state 1000000 validators, 24 modified, 0 new 890.25 ms/op 712.52 ms/op 1.25
migrate state 1000000 validators, 1700 modified, 1000 new 1.2039 s/op 901.27 ms/op 1.34
migrate state 1000000 validators, 3400 modified, 2000 new 1.4981 s/op 1.1255 s/op 1.33
migrate state 1500000 validators, 24 modified, 0 new 908.36 ms/op 694.00 ms/op 1.31
migrate state 1500000 validators, 1700 modified, 1000 new 1.2219 s/op 983.19 ms/op 1.24
migrate state 1500000 validators, 3400 modified, 2000 new 1.4286 s/op 1.0847 s/op 1.32
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 4.7300 ns/op 4.4100 ns/op 1.07
state getBlockRootAtSlot - 250000 vs - 7PWei 632.49 ns/op 383.44 ns/op 1.65
computeProposers - vc 250000 10.169 ms/op 5.6799 ms/op 1.79
computeEpochShuffling - vc 250000 135.59 ms/op 83.109 ms/op 1.63
getNextSyncCommittee - vc 250000 172.88 ms/op 96.041 ms/op 1.80
computeSigningRoot for AttestationData 27.597 us/op 29.922 us/op 0.92
hash AttestationData serialized data then Buffer.toString(base64) 2.4586 us/op 1.2464 us/op 1.97
toHexString serialized data 1.3477 us/op 789.90 ns/op 1.71
Buffer.toString(base64) 262.63 ns/op 158.06 ns/op 1.66

by benchmarkbot/action

@twoeths twoeths marked this pull request as ready for review March 19, 2024 08:24
@twoeths twoeths requested a review from a team as a code owner March 19, 2024 08:24
}
}

function writeEpochInf(dataView: DataView, offset: number, value: number): number {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@twoeths twoeths force-pushed the tuyen/validators_type branch from 3ee006b to 121edfe Compare March 26, 2024 03:42
Copy link
Member

@nflaig nflaig left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

name: "lodestar_cp_state_cache_state_persist_seconds",
help: "Histogram of time to persist state to db",
stateSerializeDuration: register.histogram({
name: "lodestar_cp_state_cache_state_serialize_seconds",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are few other metrics we could consider renaming, those are still unused (not part of dashboard)

stateReloadValidatorsSszDuration: register.histogram({

But rather part of another PR

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes will persist all metrics in a follow-up PR

@twoeths twoeths merged commit ea7c6c6 into unstable Mar 28, 2024
19 of 20 checks passed
@twoeths twoeths deleted the tuyen/validators_type branch March 28, 2024 01:29
@wemeetagain
Copy link
Member

🎉 This PR is included in v1.18.0 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants